License plate recognition system and license plate recognition method

ABSTRACT

A license plate recognition system and a license plate recognition method are provided. The license plate recognition system includes an image capturing module, a determination module and an output module. The image capturing module is utilized for capturing an image of a target object. The determination module is utilized for dividing the image of the target object into a plurality of image blocks. The determination module utilizes the plurality of image blocks to generate feature data and perform a data sorting process on the feature data to generate a first sorting result. The output module outputs the sorting result.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a recognition system and method, andmore specifically to a license plate recognition system and licenseplate recognition method.

2. Description of the Prior Art

An intersection video surveillance system is commonly used for securitymonitoring and traffic monitoring. Cameras are usually installed atintersection for recording incidents or accident occurring on the road.An operator or a security person can watch the video monitor of theintersection video surveillance system in order to maintain trafficsafety and order. However, under the long hours of monitor watchingwork, the operator or the security person may be negligent of eventoccurrence due to fatigue and stress. Thus, most of the role of theintersection video surveillance system is employed for subsequentinvestigation and obtaining corroborative evidence without for animmediate response to ongoing event or a circumstance. Moreover, whenthere is a need for tracking the trajectory of vehicle movement. Itwould require a lot of manpower and time to view the video record ofdifferent intersections.

Currently, a license plate recognition technology can be applied toactively monitor the intersection video frame. The meaningless digitalsignals of the video frame can be converted into meaningful information.However, the main factor that the technology does not become popular isthere is a need to add monitors and cameras with high image quality andlimit the install angle of cameras for meeting the size of the licenseplate. For example, the camera needs to be placed just above theintersection. Each camera is merely able to overlook one lane. Thus, itwould consume a lot of resources to establish equipment of the videosurveillance system at the beginning. Besides, the subsequentmaintenance problem also needs to be overcome.

Some factors affect the conventional license plate recognition method,such as image capturing factor and environment factor. For example,regarding the image capturing factor, the conventional license platerecognition method usually acquires the image, segments each characterof the plate number in the license plate and recognizes each characterof the plate number. The main purpose of the intersection videosurveillance system is to monitor the events occurred in or around thelane. For cost consideration, two cameras are installed at the crossroadto cover the entire intersection of the roadways. Under such acondition, since the image capturing angle becomes too large and theresolution is poor, the conventional license plate recognition methodcannot perform character segmentation operation of each license platenumber character, thus resulting in subsequent recognition errors.Besides, regarding the environment factor, the intersection videosurveillance system in outdoors may cause character segment errors dueto light and shadow changes, such as glare from the smoke and headlightsof oncoming vehicles, shadow under the shade or dark sky. Thus, theconventional license plate recognition method is unable to properlyrecognize the plate number due to the aforementioned image capturingfactor and environment factor.

SUMMARY OF THE INVENTION

Based on the aforementioned disadvantages of the prior art, it istherefore a primary objective of the present invention to provide alicense plate recognition system by using machine learning method,rather than the traditional license plate recognition systems usinghand-crafted strategies to recognize the license plate.

In order to achieve the aforementioned objective, the present inventionprovides a license plate recognition system, comprising: an imagecapturing module for capturing an image of a target object; adetermination module for dividing the image of the target object into aplurality of image blocks; and an output module; wherein thedetermination module utilizes the plurality of image blocks to generatefeature data and perform a data sorting process on the feature data togenerate a sorting result and the output module outputs the sortingresult.

According to one embodiment of the present invention, the determinationmodule comprises a vehicle detection module, a license plate detectionmodule and a license plate recognition module.

According to one embodiment of the present invention, the vehicledetection module is utilized for dividing the image of the target objectinto the plurality of image blocks, utilizing the plurality of imageblocks to generate a plurality of information and obtaining an vehicleimage through the plurality of information, the license plate detectionmodule is utilized for performing a feature determination process on thevehicle image to obtain a license plate image, and the license platerecognition is utilized for performing a feature extraction process onthe license plate image to obtain a feature vector and performing aclassifying process on the feature vector to generate correspondingprobabilities and performing the data sorting process on thecorresponding probabilities to generate the sorting result.

According to one embodiment of the present invention, the vehicledetection module comprises a grid cell division operation and thenetwork output.

According to one embodiment of the present invention, the license platerecognition module comprises a feature extraction module and a characterrecognition module.

According to one embodiment of the present invention, the featuredetermination process comprises a feature extraction, a feature mergingand an output layer.

According to one embodiment of the present invention, the presentinvention further provides a license plate recognition method,comprising: utilizing an image capturing module to capture an image of atarget object; utilizing a vehicle detection module to perform a gridcell division operation to obtain a plurality of image blocks andcalculating the plurality of image blocks to generate a plurality ofinformation and arranging the plurality of information to obtain avehicle image; and utilizing a license plate detection module to performa feature determination process on the vehicle image to obtain a licenseplate image, and utilizing a license plate recognition to perform afeature extraction process on the license plate image to obtain afeature vector, perform a classifying process on the feature vector togenerate a corresponding probability, perform a data sorting process onthe corresponding probability to generate a sorting result, andutilizing an output module to output the sorting result.

According to one embodiment of the present invention, the featuredetermination process comprises a feature extraction, a feature mergingand an output layer.

According to one embodiment of the present invention, the presentinvention further provides a license plate recognition module using themethod according to claim 8, the license plate recognition modulecomprising: a feature extraction module for performing a featureextraction operation on the license plate image to obtain a feature mapand reshapes the feature map so as to obtain a feature vector; and acharacter recognition module for classifying the feature vectors,obtaining corresponding probabilities of the feature vector accordinglyand performing a data sorting process on the corresponding probabilitiesof the feature vectors to generate a sorting result.

According to one embodiment of the present invention, the characterrecognition module comprises a long short-term memory (LSTM) and aconnectionist temporal classification (CTC).

Therefore, through machine learning method, the present inventionutilizes the intersection monitor image to obtain the license plateimage which is fed to model training. Before the model training, theimage processing technique is used to generate the license plate imagesof different sizes, angles and noise, simulate images of different sizesand resolutions, thus improving the image capturing factor. Compared toconventional license plate recognition systems using hand-craftedstrategies, the actual license plate image learned by the machinelearning method is more suitable for various environments, thusimproving the environment factor. Moreover, in order to apply to variousintersection monitors, the trained model can be regarded as the initialmodel. After obtaining a new intersection monitor image, the recognitionresult with poor confidence index can be selected through activelearning and corrected results can be obtained through manualrecognition, such that license plate recognition model can be adjustedaccordingly, thus improving the accuracy the recognition rate.

The above summary and the following detailed description andaccompanying drawings are all in order to further illustrate the presentinvention to achieve the intended purpose are taken, means and technicaleffects. Such other objects and advantages of the invention will be setforth in the subsequent description and the accompanying drawings.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a license plate recognition systemaccording to an embodiment of the present invention.

FIG. 2 is a schematic diagram of a license plate recognition moduleshown in FIG. 1 according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following illustrates specific embodiments of the present invention,and those skilled in the art can readily understand advantages andeffects of the present invention accordingly.

Please refer to FIG. 1, which is a schematic diagram of a license platerecognition system according to an embodiment of the present invention.The license plate recognition system includes an image capturing module1, a determination module 2 and an output module 3. The image capturingmodule 1 is utilized for capturing an image of a target object. Thedetermination module 2 is utilized for dividing the image of the targetobject into a plurality of image blocks, utilizing the plurality ofimage blocks to generate feature data and performing a data sortingprocess on the feature data to generate a sorting result. After that,the output module 3 outputs the sorting result. The image capturingmodule 1 includes cameras disposed at the intersection for capturingimages of vehicles passing through the intersection and the capturedimages can be accessed and provided for the following license platerecognition.

In an embodiment, the determination module 2 of the present inventionincludes a vehicle detection module 21, a license plate detection module22 and a license plate recognition module 23. The vehicle detectionmodule 21 is utilized for dividing the image of the target object intothe plurality of image blocks, utilizing the plurality of image blocksto generating a plurality of information and obtaining a vehicle imagethrough the plurality of information. The license plate detection module22 is utilized for performing a feature determination process on thevehicle image to obtain a license plate image. The license platerecognition 23 is utilized for performing a feature extraction processon the license plate image to obtain feature vectors, performing aclassifying process on the feature vectors to generate correspondingprobabilities and performing a data sorting process on the correspondingprobabilities to generate a sorting result.

In a preferred embodiment, the image of vehicle captured by the imagecapturing module 1 at the intersection is inputted to the vehicledetection module 21 of the determination module 2, such that the vehicledetection module 21 determines each frame of the inputted image of thevehicle. The vehicle detection module 21 utilizes a you only look once(YOLO) network structure to obtain the position and range of eachvehicle, compares the position of the vehicle of the previous frame andthe current frame, tracks each vehicle entering the image frame andnumbers each vehicle by using timestamp and universally uniqueidentifier (UUID).

In the YOLO network structure, the target object detection is framed asa regression problem to return the positions of the bounding boxes andassociated class probabilities. For the given image of the targetobject, the YOLO network structure simply uses the neural network onceto calculate the position of the bounding box and the probability class.The YOLO network structure includes the grid cell division operation andthe network output. The input image is divided into S×S grid cells afterperforming the grid cell division operation. Each grid cell predicts Bbounding boxes and a confidence score for the bounding boxes. Theconfidence score is the product of the probability of the object ofinterest, (Pr(Object)), and actual position of the bounding box,IOU_(pred) ^(truth). That is, the confidence score may be calculatedaccording to the following equation:confidence=Pr(Object)×IOU _(pred) ^(truth)

For the network output, each bounding box consists of 5 predictions: x,y, w, h and confidence. Where x and y represent the offset of the centerof the bounding box relative to the bounds of the grid cell, w and hrepresent the actual width and height of the bounding box relative tothe whole image. Each grid cell also predicts C conditional classprobabilities, Pr(Class_(i)|Object). Each grid cell only produces oneset of class probabilities, regardless of the number of bounding boxesB. The conditional class probabilities of each bounding box can bemultiplied with the confidence of the each bounding box to obtain aproduct result. The product result includes probability information ofthat the predicted class appearing in the bounding box, and reflects howlikely the bounding box contains an object and how accurate thecoordinate of the bounding box is. The product of the conditional classprobabilities of each bounding box and the confidence of the eachbounding box can be expressed by the following equations.confidence×Pr(Class_(i)|Object)=Pr(Class_(i))×IOU _(pred) ^(truth)

The YOLO network structure can represent a plurality of information. Theplurality of information may include 19 convolutional layers and 5 maxpooling layers. As shown in Table 1, the convolutional layer is used toextract image features. The max pooling layer is used to reduce featureparameters and preserve important features. As shown in Table 1.

TABLE 1 Type Filters Size/Stride Output Convolutional 32 3 × 3 224 × 224Max pool   2 × 2/2 112 × 112 Convolutional 64 3 × 3 112 × 112 Max pool  2 × 2/2 56 × 56 Convolutional 128 3 × 3 56 × 56 Convolutional 64 1 × 156 × 56 Convolutional 128 3 × 3 56 × 56 Max pool   2 × 2/2 28 × 28Convolutional 256 3 × 3 28 × 28 Convolutional 128 1 × 1 28 × 28Convolutional 256 3 × 3 28 × 28 Max pool   2 × 2/2 14 × 14 Convolutional512 3 × 3 14 × 14 Convolutional 256 1 × 1 14 × 14 Convolutional 512 3 ×3 14 × 14 Convolutional 256 1 × 1 14 × 14 Convolutional 512 3 × 3 14 ×14 Max pool   2 × 2/2 7 × 7 Convolutional 1024 3 × 3 7 × 7 Convolutional512 1 × 1 7 × 7 Convolutional 1024 3 × 3 7 × 7 Convolutional 512 1 × 1 7× 7 Convolutional 1024 3 × 3 7 × 7 Convolutional 1000 1 × 1 7 × 7Average pool Global 1000 Softmax

Where Filters represents the number of convolution kernel, Size/Striderepresents the size of the convolution kernel and the number of pixelsthat the filter shifts, and Output represents output pixels. Theplurality of information can be arranged so as to obtain the requiredvehicle image. In an alternative preferred embodiment, when there aremultiple vehicle images shown in the image of the target object. Therequired vehicle image can be obtained from the image with multiplevehicle images after arranging the plurality of information according tothe above-mentioned method of the vehicle detection module 21.

Further, the vehicle image is inputted to the license plate detectionmodule 22 of the determination module 2. The license plate detectionmodule 22 acquires a license plate image (picture) from the vehicleimage. For example, the license plate detection module 22 acquires thelicense plate image in the vehicle image by using an efficient andaccurate scene text detector (EAST) deep learning neural networkarchitecture. The license plate detection module 22 can choose acorresponding license plate image of the vehicle which has betterposition and resolution in the image frame for license plate recognitionthrough the UUID of the vehicle having entered into the image frame. TheEAST deep learning neural network architecture is a technique ofdetecting the position of text in the natural scene image. The EAST deeplearning neural network architecture includes two scene text detectionmethods: rotated box (RBOX) method and quadrangle (QUAD) method. A rangeof text position can be found through integrating the two scene textdetection methods. By using the RBOX method, a length, a width and arotation angle of the text bounding box of the license plate can bepredicted respectively. By using the QUAD method, four points of thetext bounding box of the license plate can be predicted. The four pointsforma quadrilateral (i.e. the text bounding box). The four points arerespectively at four corners of the quadrilateral. That is, the positionof the quadrilateral is where the text bounding box is.

The EAST deep learning neural network architecture performs a featuredetermination operation, and the feature determination operation ismainly composed of feature extraction, feature merging and output layer.The feature extraction operation extracts image features of differentresolution levels by using four convolution layers. The feature mergingoperation collects the features of different size. The feature mergingoperation merges the features. The output layer outputs the detectionresult. After that, the RBOX method is utilized to obtain the length,width and rotation angle of the text bounding box of the license plate.The QUAD method is utilized to obtain four points of the text boundingbox of the license plate, such that the license plate image is obtainedaccordingly.

Moreover, please refer to FIG. 2. FIG. 2 is a schematic diagram of thelicense plate recognition module 23 shown in FIG. 1 according to anembodiment of the present invention. After the license plate image isobtained by the license plate detection module 22, the license plateimage is transmitted to the license plate recognition module 23 of thedetermination module 2 for recognizing text in the license plate image.The license plate recognition module 23 includes a feature extractionmodule 231 and a character recognition module 232. The featureextraction module 231 performs a feature extraction operation on thelicense plate image to obtain a feature map and reshapes the feature mapso as to obtain feature vectors by using a convolutional neural network(CNN) method. The character recognition module 232 classifies thefeature vectors and obtains the corresponding probabilities of thefeature vectors by using a long short-term memory (LSTM) method andperforming a data sorting operation on the corresponding probabilitiesof the feature vectors to generate a sorting result by using aconnectionist temporal classification (CTC) method.

The license plate image, obtained by the license plate detection module22 using the EAST deep learning neural network architecture, is utilizedas an input of the feature extraction module 231 of the license platerecognition module 23. After the license plate image is transmitted tothe license plate recognition module 23, the feature extraction module231 performs image translation and mapping operations on the licenseplate image to extract txt features from the license plate image andaccordingly generate the feature map by using the CNN method. Moreover,for meeting the input requirements of the character recognition module232 of the license plate recognition module 23, the feature map can bereshaped to a set of feature vectors through mathematic translation. Thefeature vectors can be utilized as the input of the characterrecognition module 232.

The character recognition module 232 of the license plate recognitionmodule 23 receives the feature vectors corresponding to the licenseplate image from the feature extraction module 231. The feature vectorsare inputted to an LSTM network of the character recognition module 232.The LSTM network classifies the feature vector of each column (or eachrow) to predict the probability of the possible text for the featurevector of each column (or each row). The CTC method calculates a maximumprobability of a sequence prediction result according to thecorresponding probability of each feature vector and text. For example,the prediction result is “sstttt---eeeee-a----kk”. Further, the CTCmethod can remove separatrix symbols and punctuation symbols, and mergeduplicated words, such that the final prediction result is “steak”.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A license plate recognition system, comprising:an image capturing module for capturing an image of a target object; adetermination module, comprising: a vehicle detection module, fordividing the image of the target object into a plurality of imageblocks, utilizing the plurality of image blocks to generate a pluralityof information and obtaining an vehicle image through the plurality ofinformation; a license plate detection module, for performing a featuredetermination process on the vehicle image to obtain a license plateimage; and a license plate recognition module, for performing a featureextraction process on the license plate image to obtain a feature vectorand performing a classifying process on the feature vector to generatecorresponding probabilities and performing a data sorting process on thecorresponding probabilities to generate a sorting result; and an outputmodule, for outputting the sorting result.
 2. The license platerecognition system of claim 1, wherein the vehicle detection modulecomprises a grid cell division operation and the network output.
 3. Thelicense plate recognition system of claim 1, wherein the license platerecognition module comprises a feature extraction module and a characterrecognition module.
 4. The license plate recognition system of claim 1,wherein the feature determination process comprises a featureextraction, a feature merging and an output layer.
 5. A license platerecognition method, comprising: utilizing an image capturing module tocapture an image of a target object; utilizing a vehicle detectionmodule to perform a grid cell division operation to obtain a pluralityof image blocks and calculating the plurality of image blocks togenerate a plurality of information and arranging the plurality ofinformation to obtain a vehicle image; and utilizing a license platedetection module to perform a feature determination process on thevehicle image to obtain a license plate image, and utilizing a licenseplate recognition module to perform a feature extraction process on thelicense plate image to obtain a feature map, reshape the feature map toobtain a feature vector, perform a classifying process on the featurevector to generate a corresponding probability, perform a data sortingprocess on the corresponding probability to generate a sorting result,and utilizing an output module to output the sorting result.
 6. Thelicense plate recognition method of claim 5, wherein the featuredetermination process comprises a feature extraction, a feature mergingand an output layer.
 7. A license plate recognition module using themethod according to claim 6, comprising: a feature extraction module forperforming a feature extraction operation on the license plate image toobtain a feature map and reshapes the feature map so as to obtain afeature vector; and a character recognition module for classifying thefeature vectors, obtaining corresponding probabilities of the featurevector accordingly and performing a data sort process on thecorresponding probabilities of the feature vectors to generate a sortingresult.
 8. The license plate recognition module of claim 7, wherein thecharacter recognition module comprises a long short-term memory (LSTM)and a connectionist temporal classification (CTC).